Robot for preventing interruption while interacting with user

ABSTRACT

A robot includes a driver; a camera; and a processor configured to: during an interaction session in which a first user identified in an image obtained through the camera is set as an interaction subject, perform an operation corresponding to a user command received from the first user, and determine whether interruption by a second user identified in an image obtained through the camera occurs, and based on determining that the interruption by the second user occurred, control the driver such that the robot performs a feedback motion for the interruption.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is based on and claims priority under 35 U.S.C. § 119to Korean Patent Application No. 10-2019-0169741, filed on Dec. 18, 2019in the Korean Intellectual Property Office, the disclosure of which isincorporated by reference herein in its entirety.

BACKGROUND 1. Field

The disclosure relates to a robot that performs at least one operationbased on a user command, and more particularly, to a robot that preventsinterruption by another user during interaction with a specific user.

2. Description of Related Art

A related art robot that provides an interaction-based service for aplurality of unspecified users is vulnerable to interruption by anotheruser (ex. attention hijacking) during interaction with a specific user.

In particular, in the case of a robot that may be used in a public placeand/or used by a plurality of users, there is a problem that interactionwith another user may inevitably begin even before an interactionsession with a specific user is clearly finished.

SUMMARY

Embodiments provide an electronic apparatus that prevents interruptionby another user during an interaction session with a user.

Embodiments provide an electronic apparatus which, in case interruptionby another user occurs during an interaction session with a user,maintains the interaction session that is currently proceeding, andprovides a feedback motion that can prevent interruption.

A robot according to an embodiment includes a driver, a camera, and aprocessor which, during an interaction session wherein a first useridentified in an image acquired through the camera is set as aninteraction subject, performs an operation corresponding to a usercommand received from the first user. The processor is configured to,during the interaction session, determine whether interruption by asecond user identified in an image acquired through the camera occurs,and based on determining that interruption by the second user havingoccurred, control the driver such that the robot performs a feedbackmotion for the interruption.

An operating method of a robot according to an embodiment includes thesteps of, during an interaction session wherein a first user identifiedin an image acquired through a camera is set as an interaction subject,performing an operation corresponding to a user command received fromthe first user, and during the interaction session, determining whetherinterruption by a second user identified in an image acquired throughthe camera occurs, and based on determining that interruption by thesecond user having occurred, performing a feedback motion for theinterruption.

user command.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a diagram for schematically illustrating an operation of arobot according to an embodiment:

FIG. 1B is a diagram for schematically illustrating an operation of arobot according to an embodiment;

FIG. 1C is a diagram for schematically illustrating an operation of arobot according to an embodiment;

FIG. 2A is a block diagram for illustrating a configuration of a robotaccording to an embodiment of the disclosure;

FIG. 2B is a block diagram for illustrating a functional configurationof a robot according to an embodiment;

FIG. 3 is a diagram for illustrating an example wherein a robotaccording to an embodiment determines whether interruption occurs byidentifying a subject of a user command based on an image acquiredthrough a camera:

FIG. 4A is a diagram for illustrating an example wherein a robotaccording to an embodiment determines whether interruption occurs basedon the shape of input of a touch manipulation received through a touchscreen display:

FIG. 4B is a diagram for illustrating an example wherein a robotaccording to an embodiment determines whether interruption occurs basedon the shape of input of a touch manipulation received through a touchscreen display;

FIG. 5 is a diagram for illustrating an example wherein a robotaccording to an embodiment determines whether interruption occurs basedon a voice received through a plurality of microphones;

FIG. 6 is a diagram for illustrating an example wherein a robotaccording to an embodiment determines whether interruption occurs bydetermining a location of a user who is not a subject of interactionbased on an image acquired through a camera;

FIG. 7A is a diagram for illustrating a specific example wherein a robotaccording to an embodiment operates according to a user's circumstanceand whether interruption occurs after starting an interaction session;

FIG. 7B is a diagram for illustrating a specific example wherein a robotaccording to an embodiment operates according to a user's circumstanceand whether interruption occurs after starting an interaction session;

FIG. 7C is a diagram for illustrating a specific example wherein a robotaccording to an embodiment operates according to a user's circumstanceand whether interruption occurs after starting an interaction session;

FIG. 8 is a block diagram for illustrating a detailed configuration of arobot according to an embodiment;

FIG. 9 is a block diagram for illustrating an operating method of arobot according to an embodiment; and

FIG. 10 is an algorithm for illustrating an example wherein a robotaccording to an embodiment performs a feedback motion for interruptionduring an interaction session.

DETAILED DESCRIPTION

Before describing the disclosure in detail, the description format ofthis specification and the drawings will be explained.

First, as terms used in this specification and the claims, general termswere selected, in consideration of the functions of embodiments.However, the terms may vary depending on the intention of those skilledin the art, legal or technical interpretation and emergence of newtechnologies, etc. Also, there are some terms that were arbitrarilydesignated by the applicant, and the meaning of such terms may beinterpreted as defined in this specification. Meanwhile, terms that arenot specifically defined in the disclosure may be interpreted based onthe overall content of this specification and common technical knowledgein the pertinent art.

Also, the same reference numerals or symbols described in each drawingaccompanying this specification refer to components or elementsperforming substantially the same functions. For the convenience ofexplanation and understanding, the components or elements will bedescribed by using the same reference numerals or symbols in differentembodiments. That is, even if all elements having the same referencenumerals are illustrated in a plurality of drawings, the plurality ofdrawings do not mean one embodiment.

In addition, in this specification and the claims, terms includingordinal numbers such as “the first,” “the second,” etc. may be used fordistinguishing elements. These ordinal numbers are used to distinguishthe same or similar elements from one another, and the meaning of theterms are not to be interpreted in a restrictive way due to use of suchordinal numbers. For example, the orders of usage or the orders ofarrangement, etc. of elements combined with such ordinal numbers are notto be restricted by the numbers. Also, depending on needs, each ordinalnumber may be interchangeably used.

Further, in this specification, singular expressions include pluralexpressions as long as they do not mean differently in the context. Inaddition, in the disclosure, terms such as “include” and “consist of”should be construed as designating that there are such characteristics,numbers, steps, operations, elements, components or a combinationthereof described in the specification, but not as excluding in advancethe existence or possibility of adding one or more of othercharacteristics, numbers, steps, operations, elements, components or acombination thereof.

Also, in the embodiments of the disclosure, terms such as “a module,” “aunit,” and “a part” are for referring to elements performing at leastone function or operation, and these elements may be implemented ashardware or software, or as a combination of hardware and software.Further, a plurality of “modules,” “units,” and “parts” may beintegrated into at least one module or chip and implemented as at leastone processor, except when each of them needs to be implemented asindependent specific hardware.

In addition, in the embodiments of the disclosure, the description thata portion is connected to another portion includes both the case where aportion is directly connected to another portion, and the case where aportion is indirectly connected to another portion through still anothermedium. Also, the description that a portion includes an element meansthat other elements may additionally be included, but not that otherelements are excluded, unless there is any specific description meaningthe contrary.

FIG. 1A to 1C are diagrams for schematically illustrating an operationof a robot according to an embodiment.

Referring to FIG. 1A, the robot 100 may set a user 10 identified througha camera and/or a sensor, etc. as an interaction subject, and during aninteraction session wherein the user 10 is set as the interactionsubject, the robot 100 may provide one or more services to the user 10.Specifically, the robot 100 may perform one or more operations based onan instruction of the user 10 received in various forms such as a touch,a voice, a motion, etc.

Meanwhile, referring to FIG. 1B, during an interaction session with theuser 10, a user command may be input into the robot 100 from anotheruser 20. In this case, the robot 100 may determine that interruption byanother user 20 for the interaction with the user 10 occurred.

Here, referring to FIG. 1C, the robot 100 may perform a feedback motionfor avoiding or preventing interruption by the user 20. A feedbackmotion means a motion of the robot 100 for expressing refusal ordisregarding of interruption by another user during an interactionsession.

Specifically, referring to FIG. 1C, the robot 100 may move to avoid theuser 20, and may get closer to the user 10. Here, the location of therobot 100 may be changed such that the side surface or the rear surfaceof the robot 100 is toward the user 20 and the front surface of therobot 100 is toward the user 10.

As described above, the robot 100 according to an embodiment provides afeedback motion for interruption by another user who is not aninteraction subject, and thereby maintain a state wherein interactionwith a user who is an interaction subject is easy, and at the same time,express nonverbally an intent of disregarding interruption by anotheruser.

Hereinafter, a configuration and operations of a robot according to anembodiment will be described in more detail with reference to theaccompanying drawings.

FIG. 2A is a block diagram for illustrating a configuration of a robotaccording to an embodiment. Referring to FIG. 2A, the robot 100 mayinclude a driver 110, a camera 120, a processor 130, etc.

The driver 110 is a component for controlling the motions or movementsof the robot 100. For this, the driver 110 may control the moving meansof the robot 100. Other than this, the driver 110 may be electronicallyconnected with a mechanical component implementing physical movements ofthe robot 100 and drive/control the component.

Taking the robot 100 in FIG. 1A as an example, the driver 110 maycontrol mechanical components, etc. that control rotation of the wheelunder the body of the robot 100 in FIG. 1A, and the head attached on thebody of the robot 100 in FIG. 1A. Other than this, in case separatecomponents such as arms or legs are included in the robot, the driver110 may be implemented to drive the movements of the arms and legs.

The camera 120 is a component for acquiring (obtaining) an image aroundthe robot 100, and it may include an RGB camera, a depth camera, anRGB-Depth (D) camera, etc. Also, the camera 120 may be implemented as astereo camera or a 3D camera.

The robot 100 may acquire not only an RGB image but also a depth imagefor the surroundings of the robot 100 through the camera 120. In therobot 100, a plurality of cameras may be included.

The processor 130 is a component for controlling the components includedin the robot 100. The processor 130 may be implemented as ageneric-purpose processor such as a central processing unit (CPU), anapplication processor (AP), etc., a graphic-dedicated processor such asa graphic processing unit (GPU), a vision processing unit (VPU), etc.,or an artificial intelligence-dedicated processor such as a neuralprocessing unit (NPU), etc. Also, the processor 130 may include avolatile memory such as an SRAM, etc.

The processor 130 may set a user identified in an image acquired throughthe camera 120 as a subject (: a counterpart) of interaction.

Interaction may mean communication between the robot 100 and a user. Forperforming interaction with a user, the robot 100 may include a userinputter for receiving user commands, a display and a speaker foroutputting various information, etc., but the disclosure is not limitedthereto.

As an example, if a user is identified in an image acquired through thecamera 120, the processor 130 may set the user as a subject ofinteraction. Here, the processor 130 may identify the user by inputtingthe image into an artificial intelligence model trained to identifyobjects.

Also, as an example, if at least one user command is received from auser, the user may be identified through an image acquired through thecamera 120. Here, the processor 130 may set the user as a subject ofinteraction. A user command may be received at the robot 100 in variousforms such as a touch manipulation, a voice, a motion, etc.

Here, the processor 130 may detect and identify a user who input a usercommand through a sensor (a LiDAR sensor, an ultrasonic sensor, etc.)and the camera 120. Specifically, in case a user command is received,the processor 130 may detect an adjacent object through a sensor, andidentify the object (: the user) through an image photographed in thedirection of the detected object.

For example, in case a touch manipulation is input into the touch screendisplay of the robot 100, if a motion that a user's hand touches thetouch screen display is detected through an image acquired through thecamera 120, the user may be set as a subject of interaction. Also, forexample, in case a voice is received at the microphone of the robot 100,a user may be identified by using an image photographed within apredetermined range of a field of view based on the direction that themicrophone that received the voice faces, and the user may be set as asubject of interaction.

Meanwhile, after a user is identified through an image acquired throughthe camera 120, in case at least one user command is received from theidentified user, the user may be set as a subject of interaction.

As described above, if a user is set as a subject of interaction, aninteraction session with the user may start.

An interaction session means a period for the robot 100 to receive atleast one user command from a user set as a subject of interaction, andperform at least one operation based on the received user command. Also,an interaction session may mean a period for the robot 100 to transmitand receive a conversation or information with a user.

Accordingly, during an interaction session, the processor 130 mayperform an operation corresponding to a user command received from auser set as a subject of interaction. As a result, one or more servicesmay be provided to the user who is the subject of interaction. A servicemay vary according to the function of the robot 100. For example, if therobot 100 is a receptionist robot, a service that displays informationregarding a requested location or makes the robot 100 directly move tothe location according to a user command may be provided. As types ofservices may be various depending on the use of the robot 100, they arenot limited to the aforementioned examples.

During an interaction session, the processor 130 may control the driver110 such that the front surface of the robot 100 is toward the user. Onthe front surface of the robot 100, a touch screen display for receivinginput of a touch manipulation corresponding to a user command may beprovided.

Also, the processor 130 may control the driver 110 such that thelocation of the robot 100 is located within a predetermined distancefrom the user during an interaction session.

The processor 130 according to an embodiment may determine whetherinterruption by another user identified in an image acquired through thecamera 120 occurs during an interaction session.

Interruption may mean all factors that interrupt or inhibit interactionand service provision during an interaction session that is currentlyproceeding. Specifically, a user command or interception by another userduring an interaction session with a user may be examples ofinterruption, but the disclosure is not limited thereto.

Another user may mean a user who is not a user set as a subject ofinteraction. The processor 130 may distinguish a user who is a subjectof interaction and a user who is not a subject of interaction by usingthe camera 120. Here, the processor 130 may output feature informationfrom an image including the face of the user who is the subject ofinteraction, and recognize in real time the face of the user who is thesubject of interaction based on the feature information. Here, anartificial intelligence model trained to perform face recognition bycomparing feature information may be used. As a result, a user who is asubject of interaction and a user who is not a subject of interactionmay be distinguished in an image acquired through the camera 120.

Also, the processor 130 may track the location of a user who is asubject of interaction through a sensor (a LiDAR sensor, an ultrasonicsensor, etc.), and also detect another user who is not the user who isthe subject of interaction through the sensor.

In case at least one user command is received from another user duringan interaction session, the processor 130 may determine thatinterruption occurred by another user. In this regard, detailed exampleswill be described later through FIG. 3 to FIG. 5.

During an interaction session, in case it is determined that anotheruser is located between a user set as a subject of interaction and therobot 100, the processor 130 may determine that interruption occurred.In this regard, a detailed example will be described later through FIG.6.

Other than this, various events inhibiting the efficiency of interactionwith a user who is a subject of interaction may be defined to beincluded in interruption, and according to the defined interruption, theconfiguration of the robot 100 or the operation of the processor 130 canbe modified within a range of common technical knowledge.

If it is determined that interruption by another user occurred, theprocessor 130 may control the driver 110 such that the robot 100performs a feedback motion for the interruption.

A feedback motion means a motion of the robot 100 for expressing refusalor disregarding for interruption by another user during an interactionsession. Also, a feedback motion may mean a motion that expressesattention for a user set as a subject of current interaction.

For example, in case interruption by another user occurred during aninteraction session, the processor 130 may control the driver 110 toperform a feedback motion wherein the front surface of the robot 100 istoward the user set as the subject of interaction, and is againstanother user who generated the interruption.

Specifically, the processor 130 may control the driver 110 to perform afeedback motion wherein the robot 100 moves to a location wherein thefront surface of the robot is toward a user set as a subject ofinteraction and the side surface or the rear surface of the robot istoward a user who generated interruption.

Also, the processor 130 may control the driver 110 such that the robot100 approaches closer to the user set as the subject of interaction thanbefore the generation of interruption. Here, the processor 130 maycontrol the driver 110 such that the robot 100 is driven to avoid theuser who generated interruption.

Other than this, the processor 130 may control the driver 110 to performvarious feedback motions for expressing refusal for interruption.

As a result, an optimal circumstance wherein the robot 100 receives auser command of a user set as a subject of interaction and can provide aservice to the user despite interruption by another user can bemaintained.

FIG. 2B is a block diagram for illustrating a functional configurationof a robot according to an embodiment. Each module illustrated in FIG.2B may be stored in the form of software in the memory of the robot 100and may be selectively executed by the processor 130. Also, each moduleillustrated in FIG. 2B may be implemented in the form of a circuit onthe robot 100 and controlled by the processor 130. Further, each modulemay be implemented as a form wherein software and hardware are combined.

Referring to FIG. 2B, the robot 100 may include an interaction sessionmanagement module 210, a service module 220, an interruptiondetermination module 230, an interruption feedback module 240, etc.

The interaction session management module 210 is a component for settingstart/maintenance/ending of an interaction session for a user, in caseat least one user is set as a subject of interaction.

As an example, in case a user is identified in an image acquired throughthe camera 120, the interaction session management module 210 may startan interaction session with the user.

Afterwards, if a predetermined event for ending of an interactionsession occurs, the interaction session management module 210 may endthe interaction session.

The predetermined event for ending may include a case wherein provisionof a service requested by the user ends, a case wherein a user commandis not received during a specific time period from the user, etc.

Also, an event related to ending may include a case wherein it isidentified that the user does not look at the robot 100 for more than aspecific time period. In this case, the processor 130 may recognize thedirection of the user's gaze based on an image identified through thecamera 120.

The service module 220 is a module for performing at least one operationaccording to a received user command. The configuration and operation ofthe service module 220 may vary according to the function of the robot100.

Specifically, the service module 220 may control the robot 100 toperform at least one operation based on a user command received from auser set as a subject of interaction during an interaction session.

The interruption determination module 230 is a module for determiningwhether interruption by another user other than a user set as a subjectof interaction occurs during an interaction session.

As described above with an operation of the processor 130, theinterruption determination module 230 may determine whether interruptionoccurs based on a user command of another user received during aninteraction session or the location of another user during aninteraction session, etc. Detailed examples in this regard will bedescribed below through FIG. 3 to FIG. 6.

Meanwhile, the interruption feedback module 240 is a module forcontrolling the driver 110 such that the robot 100 performs theaforementioned feedback motion, in case interruption occurs during aninteraction session. As an example, the interruption feedback module 240may control the driver 110 based on motion information stored in advancein the memory of the robot 100.

Hereinafter, detailed examples wherein the robot 100 determines whetherinterruption occurs will be described through FIG. 3 to FIG. 6.

FIG. 3 is a diagram for illustrating an example wherein a robotaccording to an embodiment determines whether interruption occurs byidentifying a subject who input a user command based on an imageacquired through a camera.

Referring to FIG. 3, the robot 100 may include a display 150, e.g., atouch screen display implemented to receive input of a touchmanipulation, and a camera 120. FIG. 3 assumes a circumstance wherein aninteraction session wherein the subject of the interaction is set as auser 310 is proceeding.

Referring to FIG. 3, in case a touch manipulation is input, theprocessor 130 may determine the subject of the touch manipulation byusing an image 300 acquired through the camera 120.

Specifically, the processor 130 may identify each of a user 310 and auser 320 from the image 300, and identify whose touch manipulation thereceived touch manipulation is between the user 310 and the user 320.

Referring to FIG. 3, the processor 130 may identify that body parts 311,312 of each of the user 310 and the user 320 become close to the robot100 (or the display 150 of the robot 100) based on an image acquiredthrough the camera 120.

Specifically, the processor 130 may identify a part 321 that is includedin an area constituting the user 320 in the image 300 and is close tothe display 150 of the robot 100 based on an RGB image acquired throughthe camera 120 and depth information for each pixel corresponding to theRGB image. In this case, the processor 130 may identify that a touchmanipulation was input from the user 320.

Meanwhile, based on sensing data received through a sensor (a LiDARsensor, an ultrasonic sensor, etc.), the processor 130 may identify thata body part of the user 310 and/or the user 320 becomes close to therobot 100. According to sensing data, in case it is identified that abody part of the user 320 is close to the display 150 when a touchmanipulation is input, the processor 130 may identify that a touchmanipulation was input from the user 320.

As described above, if it is identified that an input touch manipulationis a touch manipulation of the user 320 not set as a subject ofinteraction, the processor 130 may identify that interruption occurred.

Meanwhile, in FIG. 3, only an image 300 acquired from the camera 120located on the front surface of the robot 100 was illustrated, but otherthan this, the user 320 can be identified through one or more camerasattached on various side surfaces of the robot 100.

FIGS. 4A and 4B are diagrams for illustrating an example wherein a robotaccording to an embodiment determines whether interruption occurs basedon the shape of input of a touch manipulation received through a touchscreen display. FIG. 4A and FIG. 4B illustrate areas of touches inputinto the display 150 of the robot 100 in FIG. 3.

Referring to FIG. 4A, in a state wherein a user is set as a subject ofinteraction, a plurality of touch manipulations (touch 1, 2, 3) may beinput into the display 150 from the user.

The plurality of touch manipulations in FIG. 4A may be touchmanipulations input in a state wherein it is determined that only a userwho is a subject of interaction is close to the robot 100 based on animage acquired through the camera 120.

Here, the processor 130 may identify the major axes 411, 421, 431 of thetouch areas 410, 420, 430 of each touch. Then, the processor 130 mayidentify the directions (angles) of the major axes 411, 421, 431.

Afterwards, another user who is not a subject of interaction may getclose to the robot 100. Here, the processor 130 may determine thatanother user other than a user who is a subject of interaction is alsoclose to the robot 100, based on an image acquired through the camera120.

In this case, regarding a touch manipulation that is outside thedirections of each of the major axes 411, 421, 431 of a user set as asubject of interaction by greater than or equal to a predeterminedangle, the processor 130 may identify that the touch manipulation is atouch manipulation of another user who is not a subject of interaction.

In this regard, FIG. 4B assumes a case wherein it is identified thatanother user who is not a subject of interaction is also close to therobot 100 in addition to a user who is a subject of interaction, basedon an image acquired through the camera 120.

In this case, the processor 130 may identify areas 440, 450 of each ofthe received touch manipulations (touch 4, 5), and identify thedirections (angles) of the major axes 441, 451 of each of the areas 440,450.

Referring to FIG. 4B, the direction of the major axis 441 of the area440 of the touch 4 has a big difference of about 90 degrees from thedirections of the major axes 411, 421, 431 of the touch areas 410, 420,430 of a user who is a subject of interaction. Thus, the processor 130may identify the touch 4 as a touch manipulation of another user who isnot a subject of interaction.

As a result, the processor 130 may determine that interruption occurred.

Here, the processor 130 may perform at least one function based on touchmanipulations (touch 1, 2, 3, 5) of a user who is a subject ofinteraction, but may not perform any function based on a touchmanipulation of another user (touch 4). That is, a touch manipulation ofanother user who generated interruption may be blocked.

FIG. 5 is a diagram for illustrating an example wherein a robotaccording to an embodiment determines whether interruption occurs basedon a voice received through a plurality of microphones.

FIG. 5 illustrates a circumstance wherein the robot 100 and the users510, 520 are viewed from the above. The robot illustrated in FIG. 5 maybe understood as the cross-sectional diagram of the robot 100.

FIG. 5 is based on the premise of a circumstance wherein an interactionsession wherein the user 510 is set as a subject of interaction isproceeding. Also, FIG. 5 is based on the premise of a circumstancewherein the processor 130 identified the directions of the users 510,520 through the camera 120 and/or a sensor (a LiDAR sensor, anultrasonic sensor, etc.).

Referring to FIG. 5, the robot 100 may include one or more microphones160, e.g., a plurality of microphones 160-1, 160-2, 160-3, 160-4, and160-5. Meanwhile, the arrangement state or number of the microphones arenot limited to the embodiment in FIG. 5.

Referring to FIG. 5, if a voice 521 is received from the microphones160-2, 160-3 corresponding to the direction of the user 520 among theplurality of microphones 160-1, 160-2, 160-3, 160-4, and 160-5, theprocessor 130 may determine that interruption by the user 520 occurred.

Specifically, in case it is identified that there is a voice that isreceived to be the biggest through the microphones 160-2, 160-3 locatedin the direction of the user 520 among the plurality of microphones160-1, 160-2, 160-3, 160-4, and 160-5, the processor 130 may identifythat the voice is the voice of the user 520.

Alternatively, in case the voice 521 of the user 520 is received throughthe microphone 160-2 or the microphone 160-3 simultaneously while thevoice 511 of the user 510 is being received through the microphones160-1, 160-5 toward the user 510 who is a subject of interaction, theprocessor 130 may identify that the voice of the user 520 other than theuser 510 is also being received.

Alternatively, the processor 130 may extract at least one featureinformation from the voice 511 of the user 510 and generate a model forthe voice of the user 510. Afterwards, if a voice 521 having featureinformation that is not matched with the generated model is received,the processor 130 may identify that a voice of another user who is notthe user 510 is being received.

As described above, in case a voice is received from the user 520 who isnot a subject of interaction, the processor 130 may determine thatinterruption by the user 520 occurred.

Here, the processor 130 may perform at least one function based on thevoice 511 of the user 510 who is a subject of interaction, but may notperform any function based on the voice 521 of the user 520. That is, avoice input of the user 520 who generated interruption may be blocked.

FIG. 6 is a diagram for illustrating an example wherein a robotaccording to an embodiment determines whether interruption occurs bydetermining a location of a user who is not a subject of interactionbased on an image acquired through a camera.

FIG. 6 assumes a circumstance wherein an interaction session wherein asubject of interaction is set as the user 610 is proceeding.

Referring to FIG. 6, the processor 130 may identify that the user 620who is not a subject of interaction is located between the robot 100 andthe user 610, based on the image 600 acquired through the camera 120. Inthis case, the processor 130 may determine that interruption by the user620 occurred.

Meanwhile, in case the robot 100 is implemented to perform at least onefunction based on a user motion recognized based on the camera 120 or amotion sensor, the processor 130 may perform at least one functioncorresponding to the motion of the user 610 who is a subject ofinteraction, but may not perform any function in response to the motionof the user 620 who is not a subject of interaction.

Meanwhile, the embodiments of FIGS. 3,4A, 4B, 5, and 6 may beimplemented independently from one another, but whether interruptionoccurs may be determined as a result of two or more embodiments amongthem being performed together.

FIG. 7A to FIG. 7C are diagrams for illustrating a specific examplewherein a robot according to an embodiment operates according to auser's circumstance and whether interruption occurs after starting aninteraction session.

Referring to FIG. 7A, if a user command is received from the user 710,the processor 130 may set the user 710 as a subject of interaction. Inthe case of FIG. 7A, a user command is a touch manipulation, but otherthan this, a voice or a motion, etc. are possible.

Also, unlike FIG. 7A, regardless of whether a user command was receivedfrom the user 710, if only the user 710 is identified in an imageacquired through the camera 120, the user 710 may be immediately set asa subject of interaction.

During an interaction session wherein the user 710 is set as a subjectof interaction, the processor 130 may display a user interfacecorresponding to a user command received from the user 710 on the screenof the display 150 or acoustically provide it (ex: using the speaker).

Meanwhile, as in FIG. 7B, a case wherein the user 710 who is a subjectof interaction looks away briefly or performs another personal businessduring an interaction session may occur.

In this regard, during an interaction session, the robot 100 mayidentify in real time the time that a state wherein a user command isnot received is maintained and/or the time that a state wherein thedirection of a user's gaze is not toward the robot 100 is maintained,etc.

As a specific example for identifying the direction of gaze, theprocessor 130 may recognize the head of a user by using an imageacquired through the camera 120 (ex. an RGB image). Then, the processor130 may identify points related to the directions of the face such asthe eyes, the nose, the ears, the chin, etc. in the head. Then, theprocessor 130 may recognize the direction of a user's gaze based on thedirectional relation among the identified points. Here, the processor130 may use a separate artificial intelligence model trained to detectthe direction of gaze from a facial image.

Referring to FIG. 7B, during a predetermined first time, in case a usercommand is not received from the user 710, or the user 710 is not towardthe robot 100, the processor 130 may inactivate the screen of thedisplay 150.

In case a user interface is unnecessarily displayed in a circumstancewherein the user 710 does not look at the robot 100, there is a problemthat information regarding an instruction or personal matters of theuser 710 may be unintentionally revealed to another user. Accordingly,the processor 130 may inactivate the screen in case the user 710 is nottoward the robot 100, and thereby prevent unnecessary revealing ofpersonal information.

Here, the processor 130 may control the driver 110 such that the robot100 is located within a predetermined distance 705 from the user 710. Asa result, even if the user 710 performs another job briefly, it may beexpressed clearly to the user 710 and other users that interaction withthe user 710 is still proceeding.

In a state wherein the screen of the display 150 is inactivated, in caseinterruption occurs by another user who is not the user 710 who is asubject of interaction, the processor 130 may not only control thedriver 110 to perform a feedback motion, but also display a screenindicating that an interaction session is proceeding on the display 150.

In this regard, FIG. 7C assumes a circumstance wherein interruptionoccurred by another user in the state as in FIG. 7B.

Referring to FIG. 7C, if interruption occurs, the processor 130 maycontrol the driver 110 such that the robot 100 continuously maintainsthe distance with the user 710 (within 705), and perform a feedbackmotion. In addition, the processor 130 may control the display 150 todisplay a user interface wherein a guide that ‘the robot is currentlybeing used by another user’ is written. Also, the processor 130 maycontrol the speaker of the robot 100 to output a voice 171 that ‘therobot is currently being used by another user.’

Meanwhile, during a predetermined second time longer than theaforementioned predetermined first time, in case a user command is notreceived from the user 710 who is a subject of interaction or the user710 is not toward the robot 100, the processor 130 may end theinteraction session with the user 710.

In this case, the processor 130 may set another user identified in animage acquired through the camera 120 as a subject of interaction andstart a new interaction session. Specifically, if a user command isreceived from an identified user, the processor 130 may set the user asa subject of new interaction.

FIG. 8 is a block diagram for illustrating a detailed configuration of arobot according to an embodiment.

Referring to FIG. 8, the robot 100 may further include a sensor 140, adisplay 150, a microphone 160, a speaker 170, a memory 180, etc. otherthan the driver 110, the camera 120, and the processor 130.

The sensor 140 is a component for acquiring various informationregarding the surroundings of the robot 100. The sensor 140 may includea LiDAR sensor, an ultrasonic sensor, an acceleration sensor, a gyrosensor, etc., and other than them, various sensors may be provided inthe robot 100.

As an example, the processor 130 may detect the existence of a userexisting in an adjacent location based on sensing data received througha LiDAR sensor or an ultrasonic sensor, and identify the user from animage photographed through the camera 120 in the direction of thedetected user.

Here, the processor 130 may track the user based on feature informationof an image including the user's face. Specifically, the processor 130may extract feature information of an image including the user's face,and then compare at least one area within an image consecutivelyacquired later with the feature information and thereby consecutivelytrack the location and direction of the user. Feature information may bedefined according to output of at least one artificial intelligencemodel trained to extract feature information from an image or mapfeature information with another image for face recognition.

The display 150 is a component for visually outputting various contentsor a user interface. The display 150 may be implemented as a form of atouch screen display for receiving a user command in the form of atouch.

In the robot 100, one or more displays 150 may be provided. As anexample, in case the robot consists of a body provided with a movingmeans such as a wheel and a head attached on the body, displays may beprovided respectively on the front surface of the head and the frontsurface of the body, but the disclosure is not limited thereto.

The microphone 160 is a component for receiving input of an audio signalfrom the outside. The robot 100 may include one or more microphones.

The robot 100 may perform at least one operation based on a user voicereceived through the microphone 160. Specifically, the processor 130 mayconvert a user voice into a text by using an acoustic model and alanguage model, etc., and then identify a user command corresponding tothe user voice based on the converted text. Then, the processor 130 mayoperate based on the identified user command.

The speaker 170 is a component for outputting an audio signal. Theprocessor 130 may control the speaker 170 to output various informationor a user interface in the form of a voice.

The memory 180 is a component for storing an operating system (OS) forcontrolling the overall operations of the components of the robot 100and at least one instruction or data related to the components of therobot 100.

The processor 130 may perform operations according to the aforementionedvarious embodiments by executing at least one instruction stored in thememory 180.

The memory 180 may include a non-volatile memory such as a ROM, a flashmemory, etc., and include a volatile memory consisting of a DRAM, etc.Also, the memory 180 may include a hard disc, a solid state drive (SSD),etc.

Referring to FIG. 8, the memory 180 may further include an objectrecognition module 250, a driving control module 260, etc. other thanthe interaction session management module 210, the service module 220,the interruption determination module 230, and the interruption feedbackmodule 240 described above.

The object recognition module 250 is a component for identifying atleast one object (ex. a user/a person) from an image acquired throughthe camera 120. The object recognition module 250 may use an artificialintelligence model trained to identify objects.

The driving control module 260 is a component for controlling the motionof the robot 100 through the driver 110. The driving control module 260may control the driver 110 such that a motion included in a serviceprovided to a user who is a subject of interaction or the aforementionedfeedback motion is performed by the robot 100.

Hereinafter, an operating method of a robot according to an embodimentwill be described through FIG. 9 to FIG. 10.

FIG. 9 is a block diagram for illustrating an operating method of arobot according to an embodiment.

Referring to FIG. 9, in the method according to an embodiment, anoperation corresponding to a user command received from a first user maybe performed during an interaction session wherein the first user is setas a subject of interaction at operation S910. Here, the first user maybe identified from an image acquired through the camera.

During an interaction session with the first user, in the methodaccording to an embodiment, it may be determined whether interruption bya second user identified in an image acquired through the camera occursat operation S920.

As an example, in case a user command is received from the second useridentified in an image acquired through the camera, it may be determinedthat interruption by the second user occurred.

As an example, in case the robot includes a plurality of microphones,the direction of the second user may be determined based on an imageacquired through the camera, and if a user voice is received from themicrophone corresponding to the direction of the second user among theplurality of microphones, it may be determined that interruption by thesecond user occurred.

As an example, in case the robot includes a touch screen display, if itis determined that a touch manipulation input into the touch screendisplay is a touch manipulation by the second user based on an imageacquired through the camera, it may be determined that interruption bythe second user occurred.

As an example, if it is determined that the second user is locatedbetween the robot and the first user based on an image acquired throughthe camera, it may be determined that interruption by the second useroccurred.

Further, if it is determined that interruption by the second useroccurred, in the method according to an embodiment, a feedback motionfor the interruption may be performed at operation S930.

Specifically, a feedback motion wherein the front surface of the robotis toward the first user and is against the second user may beperformed.

In case a touch screen display that is arranged on the front surface ofthe robot and is for receiving input of a touch manipulationcorresponding to a user command is provided on the robot, a feedbackmotion wherein the robot moves to a location wherein the front surfaceof the robot is toward the first user and the side surface or the rearsurface of the robot is toward the second user may be performed.

FIG. 10 is an algorithm for illustrating an example wherein a robotaccording to an embodiment performs a feedback motion for interruptionduring an interaction session.

Referring to FIG. 10, in case a user command is received from a useridentified through a camera and/or a sensor at operation S1010—Y, theuser may be set as a subject of interaction and an interaction sessionwith the user may start at operation S1020.

During an interaction session, an operation may be performed accordingto a user command received from the user at operation S1030. As aspecific example, a user interface corresponding to a user commandreceived from the user may be displayed on the screen of the touchscreen display.

While the interaction session is maintained, it may be determinedwhether interruption by another user who is not a subject of interactionoccurred at operation S1060. In case interruption occurred at operationS1060—Y, a feedback motion for refusing or disregarding the interruptionmay be performed at operation S1070.

Meanwhile, during a predetermined first time, in case a user command isnot received from a user who is a subject of interaction or the user isnot toward the robot, the screen of the touch screen display may beinactivated. Here, in case it is determined that interruption by anotheruser who is not a subject of interaction occurred, a screen notifyingthat an interaction session is proceeding may be displayed on the touchscreen display.

In case an interaction ending event occurs at operation S1040, theinteraction session may end at operation S1050. An interaction endingevent may include a case wherein an operation corresponding to a usercommand ended, a case wherein a user command is not received for morethan a specific time period, a case wherein the direction of a user'sgaze is not toward the robot for more than a specific time period, etc.

For example, during a predetermined second time longer than theaforementioned predetermined first time, in case a user command is notreceived from a user who is a subject of interaction or the user is nottoward the robot, an interaction session may end.

After an interaction session ends, in the operating method according toan embodiment, if a user command is received from a new user identifiedin an image acquired through the camera, the new user may be set as asubject of interaction and a new interaction session may start.

The operating method described above through FIG. 9 to FIG. 10 may beimplemented through the robot 100 illustrated and described through FIG.2 and FIG. 8.

Meanwhile, at least apart of the operating method described abovethrough FIG. 9 to FIG. 10 may be performed through an external apparatus(ex. a server apparatus, a terminal apparatus, etc.) that cancommunicate with the robot 100.

A robot according to an embodiment prevents interruption by another userin case an interaction session with a specific user is proceeding, andas a result of this, there is an effect that a service that is correctto the last can be provided to the original user who was being providedwith the service through the interaction session.

Also, a robot according to an embodiment has an effect that, in case auser command, etc. are received from a user who is not a subject ofinteraction while an interaction session is proceeding, it is possibleto make the user recognize that an interaction session with another useris already proceeding through a feedback motion and/or various outputs.

Meanwhile, the aforementioned various embodiments may be implemented ina recording medium that can be read by a computer or an apparatussimilar to a computer, by using software, hardware, or a combinationthereof.

According to implementation by hardware, the embodiments described inthe disclosure may be implemented by using at least one of applicationspecific integrated circuits (ASICs), digital signal processors (DSPs),digital signal processing devices (DSPDs), programmable logic devices(PLDs), field programmable gate arrays (FPGAs), processors, controllers,micro-controllers, microprocessors, or an electronic unit for performingvarious functions.

In some cases, the embodiments described in this specification may beimplemented as the processor 130 itself. Meanwhile, according toimplementation by software, the embodiments such as procedures andfunctions described in this specification may be implemented as separatesoftware modules. Each of the aforementioned software modules mayperform one or more functions and operations described in thisspecification.

Meanwhile, computer instructions for performing processing operations atthe robot 100 according to the aforementioned various embodiments of thedisclosure may be stored in a non-transitory computer-readable medium.Computer instructions stored in such a non-transitory computer-readablemedium make the processing operations at the robot 100 according to theaforementioned various embodiments performed by the aforementionedspecific machine, when the instructions are executed by the processor ofthe specific machine.

Anon-transitory computer-readable medium refers to a medium that storesdata semi-permanently, and is readable by machines, but not a mediumthat stores data for a short moment such as a register, a cache, and amemory. Specifically, the aforementioned various applications orprograms may be provided while being stored in a non-transitorycomputer-readable medium such as a CD, a DVD, a hard disc, a blue-raydisc, a USB, a memory card, a ROM and the like.

Also, while preferred embodiments of the disclosure have been shown anddescribed, the disclosure is not limited to the aforementioned specificembodiments, and it is apparent that various modifications may be madeby those having ordinary skill in the technical field to which thedisclosure belongs, without departing from the gist of the disclosure asclaimed by the appended claims. Further, it is intended that suchmodifications are not to be interpreted independently from the technicalidea or prospect of the disclosure.

What is claimed is:
 1. A robot comprising: a driver; a camera; and aprocessor configured to: during an interaction session in which a firstuser identified in an image obtained through the camera is set as aninteraction subject, perform an operation corresponding to a usercommand received from the first user, and determine whether interruptionby a second user identified in an image obtained through the cameraoccurs, and based on determining that the interruption by the seconduser occurred during the interaction session, control the driver suchthat the robot performs a feedback motion with respect to theinterruption.
 2. The robot of claim 1, wherein the processor is furtherconfigured to: control the driver to perform the feedback motion inwhich a front surface of the robot is toward the first user and is notfacing the second user.
 3. The robot of claim 2, further comprising: atouch screen display arranged on the front surface of the robot, andconfigured to receive an input of a touch manipulation corresponding tothe user command, wherein the processor is further configured to:control the driver to perform the feedback motion in which the robotmoves to a location so that the front surface of the robot is toward thefirst user and a side surface or a rear surface of the robot is towardthe second user.
 4. The robot of claim 1, wherein the processor isfurther configured to: during the interaction session, based on a usercommand being received from the second user identified in the imageobtained through the camera, determine that the interruption by thesecond user occurred.
 5. The robot of claim 4, further comprising: aplurality of microphones, wherein the processor is further configuredto: during the interaction session, determine a direction of the seconduser based on the image obtained through the camera, and based on a uservoice being received from a microphone corresponding to the direction ofthe second user among the plurality of microphones, determine that theinterruption by the second user occurred.
 6. The robot of claim 4,further comprising: a touch screen display, wherein the processor isfurther configured to: during the interaction session, based ondetermining that a touch manipulation input into the touch screendisplay is a touch manipulation by the second user based on the imageobtained through the camera, determine that the interruption by thesecond user occurred.
 7. The robot of claim 1, wherein the processor isfurther configured to: during the interaction session, based ondetermining that the second user is located between the robot and thefirst user based on the image obtained through the camera, determinethat the interruption by the second user occurred.
 8. The robot of claim1, further comprising: a touch screen display, wherein the processor isfurther configured to: based on the user command being received from thefirst user, set the first user as the interaction subject, and duringthe interaction session in which the first user is set as theinteraction subject, display a user interface corresponding to the usercommand received from the first user on the touch screen display, andduring a predetermined first time, based on the user command not beingreceived from the first user or the first user not facing the robot,inactivate a screen of the touch screen display, and control the driversuch that the robot is located within a predetermined distance from thefirst user.
 9. The robot of claim 8, wherein the processor is furtherconfigured to: while the screen of the touch screen display isinactivated, based on the determining that the interruption by thesecond user occurred, display a screen notifying that the interactionsession is proceeding, on the touch screen display.
 10. The robot ofclaim 8, wherein the processor is further configured to: during apredetermined second time longer than the predetermined first time,based on the user command not being received from the first user or thefirst user not facing the robot, end the interaction session, and afterthe interaction session for the first user ends, based on a user commandbeing received from a third user identified in an image obtained throughthe camera, set the third user as an interaction subject.
 11. Anoperating method of a robot, the operating method comprising: during aninteraction session in which a first user identified in an imageobtained through a camera is set as an interaction subject, performingan operation corresponding to a user command received from the firstuser: determining an occurrence of interruption by a second useridentified in an image obtained through the camera; and based on thedetermining the interruption by the second user during the interactionsession, performing a feedback motion with respect to the interruption.12. The operating method of claim 11, wherein the performing thefeedback motion comprises: performing the feedback motion in which afront surface of the robot is toward the first user and is not facingthe second user.
 13. The operating method of claim 12, wherein the robotincludes a touch screen display arranged on the front surface of therobot, and configured to receive an input of a touch manipulationcorresponding to the user command, and the performing the feedbackmotion further comprises: performing the feedback motion of moving to alocation in which the front surface of the robot is toward the firstuser and a side surface or a rear surface of the robot is toward thesecond user.
 14. The operating method of claim 11, wherein thedetermining the occurrence of the interruption comprises: during theinteraction session, based on a user command being received from thesecond user identified in the image obtained through the camera,determining that the interruption by the second user occurred.
 15. Theoperating method of claim 14, wherein the robot includes a plurality ofmicrophones, and the determining the occurrence of the interruptionfurther comprises: during the interaction session, determining adirection of the second user based on the image obtained through thecamera; and based on a user voice being received from a microphonecorresponding to the direction of the second user among the plurality ofmicrophones, determining that the interruption by the second useroccurred.
 16. The operating method of claim 14, wherein the robotincludes a touch screen display, and the determining the occurrence ofthe interruption further comprises: during the interaction session,based on determining that a touch manipulation input into the touchscreen display is a touch manipulation by the second user based on theimage obtained through the camera, determining that the interruption bythe second user occurred.
 17. The operating method of claim 11, whereinthe determining the occurrence of the interruption comprises: during theinteraction session, based on determining that the second user islocated between the robot and the first user based on the image obtainedthrough the camera, determining that the interruption by the second useroccurred.
 18. The operating method of claim 11, wherein the robotincludes a touch screen display, and the operating method furthercomprises: based on the user command being received from the first user,setting the first user as the interaction subject; during theinteraction session in which the first user is set as the interactionsubject, displaying a user interface corresponding to the user commandreceived from the first user on the touch screen display; and during apredetermined first time, based on the user command not being receivedfrom the first user or the first user not facing the robot, inactivatinga screen of the touch screen display.
 19. The operating method of claim18, further comprising: while the screen of the touch screen display isinactivated, based on the determining that the interruption by thesecond user occurred, displaying a screen notifying that the interactionsession is proceeding, on the touch screen display.
 20. The operatingmethod of claim 18, further comprising: during a predetermined secondtime longer than the predetermined first time, based on the user commandnot being received from the first user or the first user not facing therobot, ending the interaction session; and after the interaction sessionfor the first user ends, based on a user command being received from athird user identified in an image obtained through the camera, settingthe third user as an interaction subject.